direwolf

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 28, 2019 License: MIT Imports: 22 Imported by: 0

README

Direwolf HTTP Client: Save your time

Build Status codecov GitHub release (latest by date) language GitHub

Package direwolf is a convenient and easy to use http client written in Golang.

If you want find more info, please go here: Direwolf HTTP Client: Save your time，内有中文文档。

direwolf

Feature Support

  • Clean and Convenient API
  • Simple to Set Headers, Cookies, Parameters, Post Forms
  • Sessions Control
  • Keep-Alive & Connection Pooling
  • HTTP(S) Proxy Support
  • Redirect Control
  • Timeout Control
  • Support extract result from response body with css selector, regexp, json
  • Content Decoding
  • More to come...

Installation

go get github.com/wnanbei/direwolf

Take a glance

You can easily send a request like this:

import (
    "fmt"

    dw "github.com/wnanbei/direwolf"
)

func main() {
    resp, err := dw.Get("https://www.google.com")
    if err != nil {
        return
    }
    fmt.Println(resp.Text())
}

Besides, direwolf provide a convenient way to add parameters to request. Such as Headers, Cookies, Params, etc.

import (
    "fmt"

    dw "github.com/wnanbei/direwolf"
)

func main() {
    headers := dw.NewHeaders(
        "User-Agent", "direwolf",
    )
    params := dw.NewParams(
        "name", "wnanbei",
        "age", "18",
    )
    cookies := dw.NewCookies(
        "sign", "kzhxciuvyqwekhiuxcyvnkjdhiue",
    )
    resp, err := dw.Get("https://httpbin.org/get", headers, params, cookies)
    if err != nil {
        return
    }
    fmt.Println(resp.Text())
}

Output:

{
    "args": {
        "age": "18",
        "name": "wnanbei"
    },
    "headers": {
        "Accept-Encoding": "gzip",
        "Cookie": "sign=kzhxciuvyqwekhiuxcyvnkjdhiue",
        "Host": "httpbin.org",
        "User-Agent": "direwolf"
    },
    "origin": "1.1.1.1, 1.1.1.1",
    "url": "https://httpbin.org/get?age=18&name=wnanbei"
}

Contribute

Direwolf is a personal project now, but all contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

If you find a bug in direwolf or have some good ideas:

  • Go to GitHub “issues” tab and open a fresh issue to start a discussion around a feature idea or a bug.
  • Send a pull request and bug the maintainer until it gets merged and published.
  • Write a test which shows that the bug was fixed or that the feature works as expected.

If you need to discuss about direwolf with me, you can send me a e-mail.

Documentation

Overview

Package direwolf is a convenient and easy to use http client written in Golang.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrRequestBody = errors.New("request body can`t coexists with PostForm")
	ErrTimeout     = errors.New("reqeust timeout")
)

Functions

func WrapErr added in v0.5.2 

func WrapErr(err error, msg string) error

WrapErr will wrap a error with some information: filename, line, time and some message.

func WrapErrf added in v0.5.2 

func WrapErrf(err error, format string, args ...interface{}) error

WrapErr will wrap a error with some information: filename, line, time and some message. You can format message of error.

Types

type Body added in v0.2.0 

type Body []byte

Body is the data you want to post, one of the Request Options.

type CSSNode added in v0.4.0 

type CSSNode struct {
	// contains filtered or unexported fields
}

CSSNode is a container that stores single selected results

func (*CSSNode) Attr added in v0.4.0 

func (node *CSSNode) Attr(attrName string, defaultValue ...string) string

Attr return the attribute value of the CSSNode. You can set default value, if value isn`t exists, return default value.

func (*CSSNode) Text added in v0.4.0 

func (node *CSSNode) Text() string

Text return the text of the CSSNode. Only include straight children node text

func (*CSSNode) TextAll added in v0.4.2 

func (node *CSSNode) TextAll() string

TextAll return the text of the CSSNode. Include all children node text

type CSSNodeList added in v0.4.0 

type CSSNodeList struct {
	// contains filtered or unexported fields
}

CSSNodeList is a container that stores selected results

func (*CSSNodeList) At added in v0.4.0 

func (nodeList *CSSNodeList) At(index int) *CSSNode

At return the cssNode of specified index position. Return a empty cssNode if there is no cssNode in CSSNodeList

func (*CSSNodeList) Attr added in v0.4.0 

func (nodeList *CSSNodeList) Attr(attrName string, defaultValue ...string) (valueList []string)

Attr return a list of attribute value

func (*CSSNodeList) CSS added in v0.4.0 

func (nodeList *CSSNodeList) CSS(queryStr string) *CSSNodeList

CSS return a CSSNodeList, so you can chain CSS

func (*CSSNodeList) First added in v0.4.0 

func (nodeList *CSSNodeList) First() *CSSNode

First return the first cssNode of CSSNodeList. Return a empty cssNode if there is no cssNode in CSSNodeList

func (*CSSNodeList) Text added in v0.4.0 

func (nodeList *CSSNodeList) Text() (textList []string)

Text return a list of text. Only include straight children node text

func (*CSSNodeList) TextAll added in v0.4.2 

func (nodeList *CSSNodeList) TextAll() (textList []string)

TextAll return a list of text. Include all children node text

type Cookies 

type Cookies []*http.Cookie

Cookies is request cookies, as parameter in Request method. You should init it by using NewCookies like this: 

cookies := dw.NewCookies(
	"key1", "value1",
	"key2", "value2",
)

Note: mid symbol is comma.

func NewCookies added in v0.2.0 

func NewCookies(keyValue ...string) Cookies

NewCookies new a Cookies type.

You can set key-value pair when you init it by sent parameters. Just like this: 

cookies := NewCookies(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

func (Cookies) Add added in v0.2.1 

func (c Cookies) Add(key, value string)

Add append a new cookie to Cookies.

type Error added in v0.2.0 

type Error struct {
	// contains filtered or unexported fields
}

func (*Error) Error added in v0.2.0 

func (e *Error) Error() string

func (*Error) Unwrap added in v0.5.1 

func (e *Error) Unwrap() error

type Headers 

type Headers struct {
	http.Header
}

func NewHeaders added in v0.2.0 

func NewHeaders(keyValue ...string) *Headers

NewHeaders new a http.Header type.

You can set key-value pair when you init it by sent parameters. Just like this: 

headers := NewHeaders(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

type Params 

type Params struct {
	// contains filtered or unexported fields
}

Params is url params you want to join to url, as parameter in Request method. You should init it by using NewParams like this: 

params := dw.NewParams(
	"key1", "value1",
	"key2", "value2",
)

Note: mid symbol is comma.

func NewParams added in v0.2.0 

func NewParams(keyValue ...string) *Params

NewParams new a Params type.

You can set key-value pair when you init it by sent parameters. Just like this: 

params := NewParams(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

func (*Params) Add added in v0.2.1 

func (ssm *Params) Add(key, value string)

Add key and value to stringSliceMap. If key exists, value will append to slice.

func (*Params) Del added in v0.2.1 

func (ssm *Params) Del(key string)

Del delete the given key.

func (*Params) Get added in v0.2.1 

func (ssm *Params) Get(key string, index ...int) string

Get get the value pair to given key. You can pass index to assign which value to get, when there are multiple values.

func (*Params) New added in v0.2.1 

func (ssm *Params) New(keyValue ...string)

New is the way to create a strSliceMap. You can set key-value pair when you init it by sent params. Just like this: 

stringSliceMap{}.New(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

func (*Params) Set added in v0.2.1 

func (ssm *Params) Set(key, value string)

Set key and value to stringSliceMap. If key exists, existed value will drop and new value will set.

func (*Params) URLEncode added in v0.2.1 

func (ssm *Params) URLEncode() string

URLEncode encodes the values into “URL encoded” form ("bar=baz&foo=qux") sorted by key.

type PostForm added in v0.2.0 

type PostForm struct {
	// contains filtered or unexported fields
}

PostForm is the form you want to post, as parameter in Request method. You should init it by using NewPostForm like this: 

postForm := dw.NewPostForm(
	"key1", "value1",
	"key2", "value2",
)

Note: mid symbol is comma.

func NewPostForm added in v0.2.0 

func NewPostForm(keyValue ...string) *PostForm

NewPostForm new a PostForm type.

You can set key-value pair when you init it by sent parameters. Just like this: 

postForm := NewPostForm(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

func (*PostForm) Add added in v0.2.1 

func (ssm *PostForm) Add(key, value string)

Add key and value to stringSliceMap. If key exists, value will append to slice.

func (*PostForm) Del added in v0.2.1 

func (ssm *PostForm) Del(key string)

Del delete the given key.

func (*PostForm) Get added in v0.2.1 

func (ssm *PostForm) Get(key string, index ...int) string

Get get the value pair to given key. You can pass index to assign which value to get, when there are multiple values.

func (*PostForm) New added in v0.2.1 

func (ssm *PostForm) New(keyValue ...string)

New is the way to create a strSliceMap. You can set key-value pair when you init it by sent params. Just like this: 

stringSliceMap{}.New(
	"key1", "value1",
	"key2", "value2",
)

But be careful, between the key and value is a comma. And if the number of parameters is not a multiple of 2, it will panic.

func (*PostForm) Set added in v0.2.1 

func (ssm *PostForm) Set(key, value string)

Set key and value to stringSliceMap. If key exists, existed value will drop and new value will set.

func (*PostForm) URLEncode added in v0.2.1 

func (ssm *PostForm) URLEncode() string

URLEncode encodes the values into “URL encoded” form ("bar=baz&foo=qux") sorted by key.

type Proxy 

type Proxy struct {
	HTTP  string
	HTTPS string
}

Proxy is the proxy server address, like "http://127.0.0.1:1080". You can set different proxies for HTTP and HTTPS sites.

type RedirectError added in v0.3.0 

type RedirectError struct {
	RedirectNum int
}

func (*RedirectError) Error added in v0.5.1 

func (e *RedirectError) Error() string

type RedirectNum added in v0.2.0 

type RedirectNum int

RedirectNum is the number of request redirect allowed. If RedirectNum > 0, it means a redirect number limit for requests. If RedirectNum <= 0, it means ban redirect. If RedirectNum is not set, it means default 5 times redirect limit.

type Request 

type Request struct {
	Method      string
	URL         string
	Headers     http.Header
	Body        Body
	Params      *Params
	PostForm    *PostForm
	Cookies     Cookies
	Proxy       *Proxy
	RedirectNum int
	Timeout     int
}

Request is a prepared request setting, you should construct it by using NewRequest().

func NewRequest added in v0.5.3 

func NewRequest(method string, URL string, args ...RequestOption) *Request

NewRequest construct a Request by passing the parameters.

You can construct this request by passing the following parameters: 

method: Method for the request.
url: URL for the request.
http.Header: HTTP Headers to send.
direwolf.Params: Parameters to send in the query string.
direwolf.Cookies: Cookies to send.
direwolf.PostForm: Post data form to send.
direwolf.Body: Post body to send.
direwolf.Proxy: Proxy url to use.
direwolf.Timeout: Request Timeout.
direwolf.RedirectNum: Number of Request allowed to redirect.

type RequestOption added in v0.6.0 

type RequestOption interface {
	// contains filtered or unexported methods
}

RequestOption is the interface of Request Options. Use to bind the options to Request.

type Response 

type Response struct {
	URL           string
	StatusCode    int
	Proto         string
	Headers       http.Header
	Cookies       Cookies
	Request       *Request
	Content       []byte
	ContentLength int64
	// contains filtered or unexported fields
}

Response is the response from request.

func Delete added in v0.2.2 

func Delete(URL string, args ...RequestOption) (*Response, error)

Delete is the method to constructs and sends a Delete request. Parameters are the same with direwolf.Get()

func Get 

func Get(URL string, args ...RequestOption) (*Response, error)

Get is the most common method of direwolf to constructs and sends a Get request.

You can construct this request by passing the following parameters: 

url: URL for the request.
http.Header: HTTP Headers to send.
direwolf.Params: Parameters to send in the query string.
direwolf.Cookies: Cookies to send.
direwolf.PostForm: Post data form to send.
direwolf.Body: Post body to send.
direwolf.Proxy: Proxy url to use.
direwolf.Timeout: Request Timeout. Default value is 30.
direwolf.RedirectNum: Number of Request allowed to redirect. Default value is 5.

func Head(URL string, args ...RequestOption) (*Response, error)

Head is the method to constructs and sends a Head request. Parameters are the same with direwolf.Get()

func Patch added in v0.2.2 

func Patch(URL string, args ...RequestOption) (*Response, error)

Patch is the method to constructs and sends a Patch request. Parameters are the same with direwolf.Get()

func Post 

func Post(URL string, args ...RequestOption) (*Response, error)

Post is the method to constructs and sends a Post request. Parameters are the same with direwolf.Get()

Note: direwolf.Body can`t existed with direwolf.PostForm.

func Put added in v0.2.2 

func Put(URL string, args ...RequestOption) (*Response, error)

Put is the method to constructs and sends a Put request. Parameters are the same with direwolf.Get()

func Send added in v0.5.3 

func Send(req *Request) (*Response, error)

Send is different with Get and Post method, you should pass a Request to it. You can construct Request by use NewRequest method.

func (*Response) CSS 

func (resp *Response) CSS(queryStr string) *CSSNodeList

CSS is a method to extract data with css selector, it returns a CSSNodeList.

func (*Response) Encoding added in v0.3.0 

func (resp *Response) Encoding(encoding ...string) string

Encoding can change and return the encoding type of response. Like this: 

encoding := resp.Encoding("GBK")

You can specified encoding type. Such as GBK, GB18030, latin1. Default is UTF-8. It will decode the content to string if you specified encoding type. It will just return the encoding type of response if you do not pass parameter.

func (*Response) Re 

func (resp *Response) Re(queryStr string) []string

Re extract required data with regexp. It return a slice of string. Every time you call this method, it will transcode the Response.content to text once. So please try to extract required data at once.

func (*Response) ReSubmatch 

func (resp *Response) ReSubmatch(queryStr string) [][]string

ReSubmatch extract required data with regexp. It return a slice of string from FindAllStringSubmatch. Every time you call this method, it will transcode the Response.content to text once. So please try to extract required data at once.

func (*Response) Text 

func (resp *Response) Text() string

Text return the text of Response. It will decode the content to string the first time it is called.

type Session 

type Session struct {
	Headers http.Header
	Proxy   *Proxy
	Timeout int
	// contains filtered or unexported fields
}

Session is the main object in direwolf. This is its main features: 1. handling redirects 2. automatically managing cookies

func NewSession added in v0.2.0 

func NewSession(options ...*SessionOptions) *Session

NewSession new a Session object, and set a default Client and Transport.

func (*Session) Cookies added in v0.2.0 

func (session *Session) Cookies(URL string) Cookies

Cookies returns the cookies of the given url in Session.

func (*Session) Delete added in v0.2.2 

func (session *Session) Delete(URL string, args ...RequestOption) (*Response, error)

Delete is a post method.

func (*Session) Get 

func (session *Session) Get(URL string, args ...RequestOption) (*Response, error)

Get is a get method.

func (*Session) Head added in v0.2.2 

func (session *Session) Head(URL string, args ...RequestOption) (*Response, error)

Head is a post method.

func (*Session) Patch added in v0.2.2 

func (session *Session) Patch(URL string, args ...RequestOption) (*Response, error)

Patch is a post method.

func (*Session) Post 

func (session *Session) Post(URL string, args ...RequestOption) (*Response, error)

Post is a post method.

func (*Session) Put added in v0.2.2 

func (session *Session) Put(URL string, args ...RequestOption) (*Response, error)

Put is a post method.

func (*Session) Send added in v0.5.3 

func (session *Session) Send(req *Request) (*Response, error)

Send is a generic request method.

func (*Session) SetCookies added in v0.5.0 

func (session *Session) SetCookies(URL string, cookies Cookies)

SetCookies set cookies of the url in Session.

type SessionOptions added in v0.5.2 

type SessionOptions struct {
	// DialTimeout is the maximum amount of time a dial will wait for
	// a connect to complete.
	//
	// When using TCP and dialing a host name with multiple IP
	// addresses, the timeout may be divided between them.
	//
	// With or without a timeout, the operating system may impose
	// its own earlier timeout. For instance, TCP timeouts are
	// often around 3 minutes.
	DialTimeout time.Duration

	// DialKeepAlive specifies the interval between keep-alive
	// probes for an active network connection.
	//
	// Network protocols or operating systems that do
	// not support keep-alives ignore this field.
	// If negative, keep-alive probes are disabled.
	DialKeepAlive time.Duration

	// MaxConnsPerHost optionally limits the total number of
	// connections per host, including connections in the dialing,
	// active, and idle states. On limit violation, dials will block.
	//
	// Zero means no limit.
	MaxConnsPerHost int

	// MaxIdleConns controls the maximum number of idle (keep-alive)
	// connections across all hosts. Zero means no limit.
	MaxIdleConns int

	// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
	// (keep-alive) connections to keep per-host. If zero,
	// DefaultMaxIdleConnsPerHost is used.
	MaxIdleConnsPerHost int

	// IdleConnTimeout is the maximum amount of time an idle
	// (keep-alive) connection will remain idle before closing
	// itself.
	// Zero means no limit.
	IdleConnTimeout time.Duration

	// TLSHandshakeTimeout specifies the maximum amount of time waiting to
	// wait for a TLS handshake. Zero means no timeout.
	TLSHandshakeTimeout time.Duration

	// ExpectContinueTimeout, if non-zero, specifies the amount of
	// time to wait for a server's first response headers after fully
	// writing the request headers if the request has an
	// "Expect: 100-continue" header. Zero means no timeout and
	// causes the body to be sent immediately, without
	// waiting for the server to approve.
	// This time does not include the time to send the request header.
	ExpectContinueTimeout time.Duration

	// DisableCookieJar specifies whether disable session cookiejar.
	DisableCookieJar bool

	// DisableDialKeepAlives, if true, disables HTTP keep-alives and
	// will only use the connection to the server for a single
	// HTTP request.
	//
	// This is unrelated to the similarly named TCP keep-alives.
	DisableDialKeepAlives bool
}

func DefaultSessionOptions added in v0.5.2 

func DefaultSessionOptions() *SessionOptions

DefaultSessionOptions return a default SessionOptions object.

type Timeout added in v0.2.0 

type Timeout int

Timeout is the number of time to timeout request. if timeout > 0, it means a time limit for requests. if timeout < 0, it means no limit. if timeout = 0, it means keep default 30 second timeout.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.