How To Scrap Links from a Url in Asp.Net

How To Scrap Links from a Url in Asp.Net

Introduction

The main objective of this post is to demonstrate scraping of links from web pages(URL) using Asp.Net.
Generally, scraping of web pages is done with "HttpWebRequest" and "HttpWebResponse" method of C# in ASP.NET. However, it is observed that it becomes very difficult to fetch page data using "HttpWebRequest method. 

Background

In this post, I have created a demonstration web site. I will be scraping web site(URL) using Asp.NET.
Here, I have made use of third party tools "HtmlAgilityPack" to demonstrate this example.
You can get more information about this tool by visiting this link.

Code Snippet of Application

Aspx page:- 

<div width="100%">
        <asp:TextBox runat="server" ID="txtUrl" Width="300px"></asp:TextBox>
        <asp:Button runat="server" ID="btnscrap" Text="Scrap" OnClick="btnscrap_Click" />
        <br />
        <br />
        <asp:Label runat="server" ID="OutputLabel"></asp:Label>

    </div>


NameSpace Used Are:-

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using HtmlAgilityPack;

Note:- For NameSpace "HtmlAgilityPack" you need to add Reference in your project. You can Download  "HtmlAgilityPack" from :-

http://htmlagilitypack.codeplex.com/

cs page:- 

 protected void btnscrap_Click(object sender, EventArgs e)
        {
            var webGet = new HtmlWeb();
            var document = webGet.Load(txtUrl.Text.Trim());
            var aTags = document.DocumentNode.SelectNodes("//a");
            int counter = 1;
            if (aTags != null)
            {
                foreach (var aTag in aTags)
                {
                    if (aTag.Attributes.Contains("href"))
                    {
                        OutputLabel.Text += counter + ". " + aTag.InnerHtml + " - " + aTag.Attributes["href"].Value.ToString() + "\t" + "<br />";
                        counter++;
                    }
                }
            }

        }

Everything is done now.You just have to run the application.



Snaps:-

Enter Url And Click Download.

You will get the list of links on that url.

You can download sample code from below link:-


Scraping Example


Please give your feedback and comments on Post or You can mail me at samarthsolomon89@gmail.com ...!

Download Youtube Videos using Asp.Net

Download Youtube Videos using Asp.Net

Hi friends,

            In this post i will show you how to do download Youtube Videos using Asp.Net.


Step 1: On your aspx page paste the below code:- 

 <span>Enter YouTube URL:</span> <br />
    <asp:TextBox ID="txtYouTubeURL" runat="server" Text="" Width="450" />
    <br />
    <asp:Button ID="btnDownload" Text="Download" runat="server" OnClick="btnDownload_Click" />
    <br />
    <table>
        <tr>
            <td>
                <asp:LinkButton runat="server" ID="lnbFlV" Visible="false"></asp:LinkButton>
            </td>
        </tr>
        <tr>
            <td>
                <asp:LinkButton runat="server" ID="lnbMp4360" Visible="false"></asp:LinkButton>
            </td>
        </tr>
        <tr>
            <td>
                <asp:LinkButton runat="server" ID="lnbMp4480" Visible="false"></asp:LinkButton>
            </td>
        </tr>
        <tr>
            <td>
                <asp:LinkButton runat="server" ID="lnbMp4720" Visible="false"></asp:LinkButton>
            </td>
        </tr>
        <tr>
            <td>
                <asp:LinkButton runat="server" ID="lnbMp41080" Visible="false"></asp:LinkButton>
            </td>
        </tr>
    </table>

    <asp:Label ID="lblMessage" Text="" runat="server" />




NameSpace Used Are:-

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net;
using System.IO;
using System.Text;
using System.Text.RegularExpressions;
using System.Collections.Specialized;

using System.ComponentModel;



Step 2: Now on your cs page paste below code



 protected void btnDownload_Click(object sender, EventArgs e)
{
    DownloadYouTubeVideo();
}




private void DownloadYouTubeVideo()
        {
            try
            {
                string sURL = txtYouTubeURL.Text.Trim();
                bool isDownloaded = false;

                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(sURL);
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();

                Stream responseStream = response.GetResponseStream();
                Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
                StreamReader reader = new StreamReader(responseStream, encode);

                string pageSource = reader.ReadToEnd();
                pageSource = HttpUtility.HtmlDecode(pageSource);

                List<Uri> urls = GetVideoDownloadUrls(pageSource);

                string videoTitle = HttpUtility.HtmlDecode(GetVideoTitle(pageSource));

                foreach (Uri url in urls)
                {
                    if (url.AbsoluteUri.Contains("itag=5"))
                    {
                        lnbFlV.Text = "FLV 240p";
                        lnbFlV.PostBackUrl = url.AbsoluteUri + "&title=" + videoTitle;
                        lnbFlV.Visible = true;
                    }
                    if (url.AbsoluteUri.Contains("itag=18"))
                    {
                        lnbMp4360.Text = "MP4 360p";
                        lnbMp4360.PostBackUrl = url.AbsoluteUri + "&title=" + videoTitle;
                        lnbMp4360.Visible = true;
                    }
                    if (url.AbsoluteUri.Contains("itag=22"))
                    {
                        lnbMp4720.Text = "MP4 720p";
                        lnbMp4720.PostBackUrl = url.AbsoluteUri + "&title=" + videoTitle;
                        lnbMp4720.Visible = true;
                    }
                    if (url.AbsoluteUri.Contains("itag=137"))
                    {
                        lnbMp41080.Text = "MP4 1080p";
                        lnbMp41080.PostBackUrl = url.AbsoluteUri + "&title=" + videoTitle;
                        lnbMp41080.Visible = true;
                    }
                    if (url.AbsoluteUri.Contains("itag=135"))
                    {
                        lnbMp4480.Text = "MP4 480p";
                        lnbMp4480.PostBackUrl = url.AbsoluteUri + "&title=" + videoTitle;
                        lnbMp4480.Visible = true;
                    }

                }


            }
            catch (Exception e)
            {
                lblMessage.Text = "An error occurred while downloading video: " + e.Message;
            }
        }



private List<Uri> GetVideoDownloadUrls(string pageSource)
        {
            string pattern = "url=";
            string queryStringEnd = "&quality";

            List<Uri> finalURLs = new List<Uri>();
            List<string> urlList = Regex.Split(pageSource, pattern).ToList<string>();

            foreach (string url in urlList)
            {
                string sURL = HttpUtility.UrlDecode(url).Replace("\\u0026", "&");

                int index = sURL.IndexOf(queryStringEnd, StringComparison.Ordinal);
                if (index > 0)
                {
                    sURL = sURL.Substring(0, index).Replace("&sig=", "&signature=");
                    finalURLs.Add(new Uri(Uri.UnescapeDataString(sURL)));
                }
            }

            return finalURLs;
        }



private string GetVideoTitle(string pageSource)
        {
            int startIndex = pageSource.IndexOf("<title>");
            int endIndex = pageSource.IndexOf("</title>");

            if (startIndex > -1 && endIndex > -1)
            {
                string title = pageSource.Substring(startIndex + 7, endIndex - (startIndex + 7));
                return title;
            }

            return "Unknown";

        }

Everything is done now. Run the application.



Snaps:-

Enter Youtube Url And Click Download.

Now Links Will Appear Click on Quality You Want To Download

It will prompt to save the file

You can download sample code from below link:-


Youtube Video Download Example


Please give your feedback and comments...!

Simple Paging and Sorting on GridView in Asp.net

Simple Paging and Sorting on GridView in Asp.net 

Hi friends,

            In this post i will show you how to do paging and sorting on gridview. 

Step 1: On your aspx page paste the below code:-

<asp:GridView runat="server" ID="gv_records" AllowPaging="True" AllowSorting="True"

        AutoGenerateColumns="False" OnPageIndexChanging="gv_records_PageIndexChanging"

        OnRowDataBound="gv_records_RowDataBound" OnSorting="gv_records_Sorting" PagerStyle-BackColor="Orange" PageSize="8"

        PagerStyle-ForeColor="Black" PagerStyle-Font-Underline="false" PagerSettings-Mode="Numeric" PagerStyle-HorizontalAlign="Center" >

        <Columns>

            <asp:BoundField DataField="CountryCode" HeaderText="CountryCode" SortExpression="CountryCode"

                HeaderStyle-BackColor="Orange" HeaderStyle-ForeColor="Black" HeaderStyle-Font-Underline="false" />

            <asp:BoundField DataField="CountryName" HeaderText="CountryName" SortExpression="CountryName"

                HeaderStyle-BackColor="Orange" HeaderStyle-ForeColor="Black" HeaderStyle-Font-Underline="false" />

        </Columns>

    </asp:GridView>

Step 2: Now on your cs page paste below code above Page_Load

 public string sortingOrder  // For Sort Order

        {

            get

            {

                if (ViewState["sortingOrder"] == null)

                {

                    ViewState["sortingOrder"] = "desc";

                }

                return ViewState["sortingOrder"].ToString();

            }

            set

            {

                ViewState["sortingOrder"] = value;

            }

        }

        public string SortColumn   // For Sort Column

        {

            get

            {

                return (ViewState["SortColumn"] == null ? "CountryCode" : ViewState["SortColumn"].ToString()); // CountryCode is column name

            }

            set

            {

                ViewState["SortColumn"] = value;

            }

        }

        public SortDirection sortDirection   // For Sort Direction

        {

            get

            {

                if (ViewState["sortdirection"] == null)

                {

                    ViewState["sortdirection"] = SortDirection.Ascending;

                    //return SortDirection.Ascending;

                }

                return (SortDirection)ViewState["sortdirection"];//.Descending;

            }

            set

            {

                ViewState["sortdirection"] = value;

            }

        }

Step 3: Now write a method for getting data from database in DataSet.

In my case i have written a method fillgrid() to fill gridview and ExecuteSPDataSet() to get data in DataSet .

protected void fillgrid()

        {

            DataSet ds = ExecuteSPDataSet("usp_getcountry"); // This code is for getting data in DataSet.

// Below code is getting data from DataSet in DataView for sorting and binding to GridView. 

            DataView dv = new DataView(ds.Tables[0]);

            dv.Sort = SortColumn + " " + sortingOrder;

            gv_records.DataSource = dv;

            gv_records.DataBind();

        }

Now we have to call above method in Page_Load so write the below code in Page_Load

 if (!IsPostBack)

            {

                fillgrid();

            }

Now paste the below code:-

 protected void gv_records_PageIndexChanging(object sender, GridViewPageEventArgs e)

        {

            gv_records.PageIndex = e.NewPageIndex;

            fillgrid();

        }

        protected void gv_records_Sorting(object sender, GridViewSortEventArgs e)

        {

            if (sortDirection == SortDirection.Descending)

            {

                sortDirection = SortDirection.Ascending;

            }

            else

            {

                sortDirection = SortDirection.Descending;

            }

            sortingOrder = (sortDirection == SortDirection.Descending) ? "asc" : "desc";

            SortColumn = e.SortExpression;

            

            fillgrid();

        }

        protected void gv_records_RowDataBound(object sender, GridViewRowEventArgs e)

        {

            string imgAsc = @" <img src='../Images/white-triangle-up.png' Width='10px' Height='10px' border='0' title='Ascending' />";

            string imgDes = @" <img src='../Images/white-triangle-down.png'Width='10px' Height='10px' border='0' title='Descendng' />";

            if (e.Row.RowType == DataControlRowType.Header)

            {

                foreach (TableCell cell in e.Row.Cells)

                {

                    if (cell.HasControls())

                    {

                        LinkButton lbl = (LinkButton)cell.Controls[0];

                        if (lbl.Text == SortColumn)

                        {

                            if (sortingOrder == "asc")

                            {

                                lbl.Text += imgAsc;

                            }

                            else

                            {

                                lbl.Text += imgDes;

                            }

                        }

                    }

                }

            }

        }

Everything is done now. Run the application.

Paging:-

Sorting:-

You can download sample code from below link:-

Paging Sorting Example

Please leave your feedback and comments below..!

User Control in Asp.Net

User Control in Asp.Net


Hi Friendz,
           In this post i will show you how to use user control.

First you have to know what is user control.

In addition to using Web server controls in your ASP.NET Web pages, you can create your own custom, reusable controls using the same techniques you use for creating ASP.NET Web pages. These controls are called user controls.

A user control is a kind of composite control that works much like an ASP.NET Web page—you can add existing Web server controls and markup to a user control, and define properties and methods for the control. You can then embed them in ASP.NET Web pages, where they act as a unit.

Now we will see how to create a user control and use it.

Step 1: Right Click on your Project and add new item a popup will appear. Select  Web User Control and Name it Registration.

Step 2: Open Registration.ascx and paste below code.

<div id="RegUserControl">

    <div id="RegUserControl_header">

        <h1 style="color: Black;">

            Free Registration Form!</h1>

    </div>

    <hr style="margin-top: 0px; margin-left: 1px; width: 223px;" />

    <br />

    <asp:Table ID="Table1" runat="server" Style="margin-left: 5px; color: Black;">

        <asp:TableRow>

            <asp:TableCell>

                    <input type="text" id="txtFirstName" placeholder="First Name" style="width: 94px;font-size:10px;" />

            </asp:TableCell>

            <asp:TableCell>

                    <input type="text" id="txtLastName" placeholder="Last Name" style="width: 94px;font-size:10px;" />

            </asp:TableCell>

        </asp:TableRow>

        <asp:TableRow>

            <asp:TableCell>

                    <select id="ddSelectProfile" style="width: 99px;font-size:10px;">

                        <option value="-1">Select Profile</option>

                        <option value="Trader">Trader</option>

                        <option value="Investor">Investor</option>

                        <option value="Employed">Employed</option>

                    </select>

            </asp:TableCell>

            <asp:TableCell>

                    <select id="ddSelectCity" style="width: 99px;font-size:10px;">

                        <option value="-1">Select City</option>

                        <option value="saab">Las Vegas</option>

                        <option value="mercedes">New York</option>

                        <option value="audi">London</option>

                    </select>

            </asp:TableCell>

        </asp:TableRow>

        <asp:TableRow>

            <asp:TableCell>

                    <input type="text" id="txtMobPrfix" value="+91" disabled="disabled" style="width: 24px;font-size:10px;" />

                    <input type="text" id="txtMobNo" style="width: 59px;font-size:10px;" placeholder="Phone" />

            </asp:TableCell>

            <asp:TableCell>

                    <input type="text" id="txtEmail" placeholder="Email" style="width: 93px;font-size:10px;" />

            </asp:TableCell>

        </asp:TableRow>

        <asp:TableRow>

            <asp:TableCell ColumnSpan="2">

                    <textarea id="txtRequirments" style="width: 198px; height: 35px;font-size:10px;font-family:Arial; resize: none;" placeholder="Please describe Your Requirments."></textarea>

            </asp:TableCell>

        </asp:TableRow>

        <asp:TableRow>

            <asp:TableCell ColumnSpan="2">

                    <div style="width: 200px; font-size: 10px;font-family: Arial;">

                        <input type="checkbox" id="chkT&C" />I agree to all terms & conditions.<span style="color: red;">*</span></div>

            </asp:TableCell>

        </asp:TableRow>

        <asp:TableRow>

            <asp:TableCell ColumnSpan="2">

                    <div style="width: 200px; font-size: 10px;">

                        <input type="submit" style="background-color:#11B5E7;border-radius: 5px;-moz-border-radius: 5px;-webkit-border-radius: 5px;color:White;" 

                        id="btnSubmit" value="Submit" /></div>

            </asp:TableCell>

        </asp:TableRow>

    </asp:Table>

</div>

Step 3: Open the page where you want to add user control.After the page tag paste the below code as shown in the image.

<%@ Register TagPrefix="sam" TagName="Label" Src="~/Registration.ascx" %>

Step 4: Now Paste the below code in your MainContent 

 <sam:Label runat="server" ID="Reg_usercontrol" Title="Regestration" />

Now run the project you will see the user control 

Now any where in the project you want this user control just follow Step 3 & Step 4.

You can download sample code from below link:-

User Control Example

Please leave your feedback and comments below..!

AJAX Cascading Dropdown in Asp.NET

AJAX Cascading Dropdown in Asp.NET

Hi Friends,

           In this post i have shown how to use AJAX Cascading Dropdown in Asp.Net. Also added sample Code at the end of the post.

Step 1: Download the ajax toolkit from Ajax Toolkit and extract it.

Step 2: Now open toolbox in Visual Studio then Right Click on it and Add Tab

Now enter tab Name and press Enter Key

Now Right Click on your New tab And Select Choose Items...

A popup box will appear Click on Browse then go to the path where you have extracted AjaxToolkit and select AjaxControlToolkit.dll then press OK

Now you will see the list of ajax tools

Now we are Ready to use Ajax Cascading Dropdown

Now we have to add reference in the project 

Note:- First check references if AjaxControlToolkit is there then no need to add reference Proceed to Step 3

Right Click on References in your project then click Add Reference

A popup box will appear click on browse tab then go to folder where you have extracted ajax toolkit and select AjaxControlToolkit.dll then press OK

Step 3: Now Create a new WebForm and paste the below code

<div>

        <table align="center">

            <tr>

                <td colspan="2">

                    <b>Cascading Dropdown Sample</b>

                </td>

            </tr>

            <tr>

                <td>

                    Select Country:

                </td>

                <td>

                    <asp:DropDownList ID="ddCountry" runat="server" Width="200px">

                    </asp:DropDownList>

                    

                </td>

            </tr>

</table>

</div>

Now Drag and Drop Cascading DropDown From Tool Box Near asp:dropdownlist

Step: 4 Now Right Click on your project and add new item to your project 

then select Web Service and click add button

Step: 5 Open WebService1.asmx.cs and add the name space

using AjaxControlToolkit;

Now paste below code:-

 [WebMethod]

        public CascadingDropDownNameValue[] BindCountry(string knownCategoryValues, string category)

        {

            List<CascadingDropDownNameValue> countrydetails = new List<CascadingDropDownNameValue>();

            countrydetails.Add(new CascadingDropDownNameValue("India", "1"));

            countrydetails.Add(new CascadingDropDownNameValue("USA", "2"));

            countrydetails.Add(new CascadingDropDownNameValue("Germany", "3"));

            countrydetails.Add(new CascadingDropDownNameValue("South Africa", "4"));

            countrydetails.Add(new CascadingDropDownNameValue("Russia", "5"));

            countrydetails.Add(new CascadingDropDownNameValue("China", "6"));

            countrydetails.Add(new CascadingDropDownNameValue("Japan", "7"));

            countrydetails.Add(new CascadingDropDownNameValue("Canada", "8"));

            countrydetails.Add(new CascadingDropDownNameValue("Rome", "9"));

            countrydetails.Add(new CascadingDropDownNameValue("Brazil", "10"));

            return countrydetails.ToArray();

        }

Now your page should look like this:-

Now Un-comment this line:- 

    [System.Web.Script.Services.ScriptService]

Step: 5 Now in cascading dropdown on your web form add the below code after runat="server"

Category="Country"

                    TargetControlID="ddCountry" LoadingText="Loading Countries..." PromptText="Select Country"

                    ServiceMethod="BindCountry" 

after ServiceMethod paste this :- 

 ServicePath="~/WebService1.asmx"

Now it Should look like this :-

 Step: 6 Now drag and Drop toolkitscriptmanager from toolbox after form tag 

  

Now you are Done Press F5 to start debugging and you will see your dropdown

You can download sample code from below link:-

Cascading Dropdown Example

Please leave your feedback and comments below..!